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(57) Abstract 



The position of a pointer (24) on a 
computer display (14) may be controlled using 
captured streaming video. One or more digital 
cameras (16, 18, 20) may record actions 
inputted by the user. The recorded images 
are analyzed using appropriate software (52), 
such as pattern recognition software, to detect 
a particular object shape which corresponds 
to a given cursor command. When this 
object shape is detected, its direction, extent 
and/or rate of movement may be analyzed. 
Information about the direction, extent and/or 
rate of movement of the object may then be 
used to control the position on the pointer on 
a display screen. 
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Controlling A Pointer Using Digital Video 

Background 

This invention relates generally to computers and particularly to pointing devices 
which control the position of display pointers or cursors. 

A variety of pointing devices have been used in computer systems to control the 
position of a pointer or cursor on a display screen. The pointer or cursor is an image that may 
be moved across the screen to select a particular option provided in the form of a physical 
depiction on a user interface. In this way, a user may interact with a video display to provide 
focus to particular tasks and activities offered through the user interface. 

Well known pointing devices include the mouse, which is held in the hand and moved 
over a surface. Movements of the mouse result in corresponding movements of the pointer 
through a mouse interface which converts the mouse movements into pointer display control 
signals. 

Another familiar pointing device is the track ball which is essentially an inverted 
mouse. The track ball is a rotary ball which can be moved by the user. Movements of the 
ball are sensed and result in corresponding movements of the pointer. Another known 
pointing device is the pointing stick which is provided in the middle of the keyboards of 
certain computers. The stick may be tilted to control the position of the pointer. 

Touch pad pointing devices allow the user to touch a contact surface. Movements of 
the user's finger on the contact surface are tracked as corresponding movements of the 
pointer on the computer display. Normally touch pads have capacitive sensors which detect 
finger position and movement and translate this movement into pointer position control 
signals. 

Existing pointing devices generally involve an electromechanical interface which is 
prone to failure. In addition, the interface may become fouled, resulting in incorrect signals 
which may cause the movement of the pointing device to be incorrectly reflected in the 
pointer position on a display. In many cases, the extent of available movement of the 
pointing device is limited, making it difficult to learn to control the pointer using the limited 
movements available through the pointing device. 

Thus there is a need for an improved pointing device which overcomes the 
disadvantages inherent in the existing pointing devices. 
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Summary 

In accordance with one embodiment, a method for enabling control of a computer 
pointer includes enabling a computer to receive a digital video stream. A way is provided to 
analyze the video stream. A connection is created that allows the analysis of the video stream 
to be used to control the pointer position. 

Brief Description of the Drawings 

Figure 1 is a side elevational view of one embodiment of the present invention; 

Figure 2 is a perspective view of the embodiment shown in Figure 1 ; 

Figure 3 is a block depiction of the embodiment shown in Figures 1 and 2; 

Figure 4 is a diagram showing a method for implementing video control over pointer 
position; 

Figure 5 is a flow diagram for software which may be used to control pointer position 
using one or more digital video cameras; and 

Figures 6 and 7 are flow diagrams for calibration software. 

Detailed Description 

A computer system 10, shown in Figure 1, may include a keyboard base 12 and a 
display 14. The computer system 10 is illustrated as a portable computer system but other 
computer systems including desktop computer systems may be used in connection with the 
present invention as well. 

A plurality of digital video cameras 16, 18 and 20, as shown in Figures I and 2, may 
be coupled to the computer system 10. For example the cameras may be mounted on the 
display 14. The cameras may capture images created by the user for the purpose of 
controlling the position of a pointer 24 on the screen 22 of the display 14. 

For example, the user's hand A may assume a finger pointing position as shown in 
Figures 1 and 2. The hand A may be moved relative to the computer. The particular pattern 
of the pointed finger may be one which is recognized by software contained in the computer 
as indicating that pointer control is desired. The digital cameras 16, 18 and 20 may capture 
the image of the hand A and its movement, indicated by the arrow B, and may reflect that 
movement in corresponding movements of the pointer 24, indicated by the arrow C in Figure 
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While the present invention has been illustrated with respect to a simple finger 
pointing embodiment, other captured images may be recognized by the computer system as 
pointer control inputs. For example, a closed hand, a pointer of some particular shape in the 
hand, and the orientation and movement of the user's face could all be recognized as pointer 

5 control commands as well. Also, the motion of an object may be used to indicate direction, 
speed and other information. 

With additional semantic processing one could translate a motion into its underlying 
meaning. For example, a head nodding yes or shaking no could be translated into a button 
click of Yes or No (or OK or Cancel) when a dialog box is on the screen. Similarly, hand 

1 0 gestures like a "thumbs up" for Yes and a sharp wave of the hand for No could be used. 

In this way, the pointer position may be controlled without requiring actual contact 
with the computer itself. This eliminates the necessity for electromechanical connections 
used with the mouse and other pointing devices. Thus, it may be possible to improve the 
accuracy of the translation of the input pointer control commands to pointer movement. 

15 Referring now to Figure 3, an exemplary computer system 10 for implementing an 

embodiment of the present invention includes a processor 26 communicating with a bridge 28 
and system memory 30. The bridge 28 may couple a bus 32 which in turn couples a display 
controller 34, a video interface 36 and another bridge 38. The display controller 34 may 
couple the display screen 22. The video interface 36 may provide a connection to the 

20 cameras 16, 1 8 and 20. The bridge 38 may couple another bus (not shown) and a hard disk 
drive 40 which may store a software program 52 thereon. The software programs 52, 78 and 
79 may be loaded into the system memory and executed. 

Referring to Figure 4, the ability to control the pointer position using digital video 
requires that the computer system be enabled to receive a digital video stream (block 72). In 

25 addition an ability to analyze the video stream (block 74) must be provided as well. 

Conventionally, this video analysis may involve well known pattern recognition algorithms 
which may be preprogrammed at the factory or may be implemented through learning 
software. The learning software allows the user to train the software to recognize a particular 
user's fingers, movements, or desired attributes and to recognize them as particular pointer 

30 control commands. 

Next," the computer system may be provided with a connection (block 76) which — - 
allows the analysis of the video signal to be used to control pointer position. To this end, 
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there may be a suitable connection implemented by a computer system which allows the 
results of the pattern recognition analysis to be communicated to a display controller 34 and a 
display screen 22. These steps may be implemented by the computer system manufacturer or 
by the developer of software such as application software which implements the pointing 
5 device functionality. This functionality could be provided in the form of software operated 
by the computer system 10 or it could be provided by a peripheral such as one of the digital 
cameras 16, 18 or 20. 

As shown in Figure 5, the software 52 for controlling pointer position begins upon the 
receipt of streaming video (block 54) from one or more of the digital video cameras 16-20. 

10 In one embodiment, a frame of the streaming video may be selected (block 56) for analysis. 
The frame may be selected randomly, or at regular intervals or using a selection algorithm. 

In addition, a scene change detector may be used to determine when there is a 
significant change in scene. When a significant scene change occurs, a frame may be 
captured for subsequent analysis. Scene change detectors are known in the art and may 

1 5 involve an analysis of the pattern of brightness signals determined by the pixels that make up 
an imaging array inside one or more cameras. For example, the various pixel sensors may 
provide outputs. If an output is above a certain level it may be considered white and if it is 
below a certain level it may be deemed to be black. A significant change in the number of 
white and black outputs can be recognized as a scene change initiating the selection of one 

20 frame from the streaming video. 

While the present invention has been described with respect to the use of a single 
frame, a continuing analysis could be done by successively or continually analyzing 
successive frames. 

Once a frame is selected, as indicated in block 56, conventional pattern recognition 
25 software may be utilized (block 58) to determine if a particular pattern is reflected by the 
digital video data stream. This software may recognize a particular preprogrammed image 
such as the finger pointing image or it may be trained to leam a particular image through 
cooperation of the computer owner or user. 

Alternatively, a distinct object may be associated with the user so that the object may 
30 be easily recognized. For example, a distinctly colored ring may be used which can be 
readily detected. Movement of the ring may be detected as a pointer position command; 
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At diamond 60, the software checks to determine whether the selected frame contains 
a pattern matching a predetermined pattern which corresponds to a pointer position 
command. If not, the flow returns to select another frame. 

If a pattern match is detected, the first and subsequent frames are compared (block 62) 
5 and a determination is made as to whether there has been a change in position (block 66). If a 
position change is recognized, a rate and direction of movement of the image are determined. 
The extent, rate and/or direction of movement can be used to provide a corresponding 
movement of the pointer (block 67). This is done by sending a signal to the display controller 
to move the pointer (block 68) on the display screen 22 according to the extent, direction or 
10 rate. 

The software 78 for enabling the system to calibrate a focus point for pointer position 
commands is shown in Figure 6. The software 78 may be utilized during a calibration period 
in order to program the system to recognize the desired inputs. Referring to Figure 6, initially 
streaming video is received as indicated in block 80. A frame of that streaming video is 

1 5 captured, as illustrated in block 82. 

Next, the system is prompted for movement in a defined direction as shown in block 
84. For example, an onscreen display may guide the user through the learning process. For 
example, the onscreen display may first direct the user to show a left cursor position 
command. The sequence then cycles through each of the cursor command directions. Once 

20 the system knows what cursor command is being programmed, it may capture a new 

(subsequent) frame of video as indicated in block 86. The new (subsequent) frame and the 
prior frame are compared to determine a difference that amounts to a given cursor command 
(block 88). A check at diamond 90 determines whether a focus point has been identified. In 
other words, the system must be able to differentiate the two frames sufficiently to be able to 

25 thereafter recognize those differences as a particular pointer position command. If not, the 
process is tried again. Otherwise, the system saves off the focus point data of the identified 
focus point, as indicated in block 92. 

Referring to Figure 7, the software 79 enables a calibration of speed and direction of 
pointer position commands. The focus point data of the focus point determined by the 

30 software 78 is loaded at block 94. Streaming video is received as indicated at block 96 and a 
frame" is captured as shown in block 987 A focus point is identified'stich as the user 'sTfrigef 
as indicated in block 100 and a movement prompt is provided (block 102). The prompt may 
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be a message block on the screen asking the user to implement what the user wants to use as 
a desired movement. A new (subsequent) frame is captured in block 104 and compared to 
the frame captured in block 98, as illustrated by block 106. 

The focus point movement is identified (block 108) and a direction and rate of 
5 movement are calculated as indicated in block 1 10. A correlation factor for x, y and z 

directions is used to calculate the corresponding rate and direction of movement of the screen 
pointer (block 112). The screen pointer is then moved as indicated in block 114. The user 
then may adjust the amount of pointer movement for a given movement of the user's finger, 
for example through inputs provided by a prompt screen, as indicated in block 116. In 

10 addition, the user may provide alternative inputs which could be received as a particular 

pointer position command. The user is then prompted to indicate whether the user is done in 
diamond 118. If not, the flow cycles and new commands may be learned as described above. 
Otherwise the flow is terminated and the information is saved (block 120). 

The number of cameras that may be used may be subject to considerable variation. 

1 5 More cameras may result in better 3D imaging. This may allow movements in three 

dimensions to be recognized as pointer command signals. In addition, camera redundancy 
may be used to eliminate errors. 

While the present invention has been illustrated with respect to a separate camera 
mounted on the computer, the camera could be integrated into the housing of the computer 

20 system itself. In addition, while the invention has been illustrated in connection with cameras 
sensitive to light in the visible spectrum, infrared detecting cameras could be used as well. 

By using streaming video to control pointer position, an electromechanical connection 
(which may be prone to error or failure) may be eliminated. In addition, the accuracy of the 
pointing device may be improved, for example, as compared to touch pad pointing devices, 

25 because the possibility of incorrect actuation due to dirt or moisture may be eliminated. 

While the present invention has been described with respect to a limited number of 
embodiments, those skilled in the art will appreciate numerous modifications and variations 
therefrom. It is intended that the appended claims cover all such modifications and variations 
as fall within the true spirit and scope of the present invention. 

30 What is claimed is: 
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1 1 . A method for controlling a computer pointer comprising: 

2 receiving a digital video stream; 

3 analyzing said digital video stream; and 

4 controlling the position of the pointer based on the analysis of said video 

5 stream. 

1 2. The method of claim 1 wherein said step of receiving a digital video stream 

2 includes capturing a digital video image and inputting a representation of said image to said 

3 computer. 

1 3. The method of claim 1 wherein said analyzing step involves using pattern 

2 recognition software to analyze said digital video stream. 

1 4. The method of claim 1 including extracting a first video frame, and analyzing 

2 said video frame using pattern recognition software. 

1 5. The method of claim 4 including extracting a subsequent frame and comparing 

2 said first video frame and subsequent frames. 

1 6. The method of claim 1 including determining a direction of movement of an 

2 object depicted in said digital video stream. 

1 7. The method of claim 6 determining the rate of movement of said object. 

1 8. The method of claim 7 including detecting a scene change and when a scene 

2 change is detected, selecting a new frame and analyzing said new frame. 

1 9. An article comprising a medium for storing instructions that cause a computer 

2 to: 

3 receive a digital video stream; 

4 analyze said digital video stream; and 

5 control the position of said pointer based on the analysis of said video stream. 
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1 10. The article of claim 9 wherein including instructions that cause a computer to 

2 recognize a pattern corresponding to a particular object from said digital video stream. 

1 11. The method of claim 1 further including: 

2 detecting a gesture having a semantic meaning; and 

3 converting said detected gesture into an input for an electronic device; 

1 12. The method of claim 1 1 wherein converting includes converting said gesture 

2 into a pointer control signal. 

1 13. The method of claim 1 1 wherein detecting a gesture includes detecting 

2 gestures having the semantic meaning of yes or no. 

1 14. The method of claim 1 1 wherein detecting includes developing a video frame 

2 of said gesture. 

1 15. The method of claim 24 further including using pattern recognition software to 

2 detect said gesture. 

1 16. A system comprising: 

2 a processor; 

3 memory associated with said processor, said memory storing a computer 

4 program that causes said processor to: 

5 receive a digital video stream; 

6 analyze said digital video stream; and 

7 control the position of a pointer based on the analysis of said video 

8 stream. 



17. The system of claim 6 including a plurality of digital video cameras. 
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